{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visual Place Recognition: A Tutorial\n",
    "This notebook shows a practical code example in Python that illustrates to prospective practitioners and researchers how VPR is implemented and evaluated. The example implements a basic VPR pipeline with the key steps and components that are part of most VPR pipelines.\n",
    "\n",
    "For details or if you use our work for your academic research, please refer to the following paper:\n",
    "```bibtex\n",
    "@article{SchubertVisual,\n",
    "    title={Visual Place Recognition: A Tutorial},\n",
    "    author={Stefan Schubert and Peer Neubert and Sourav Garg and Michael Milford and Tobias Fischer},\n",
    "    journal={arXiv 2303.03281},\n",
    "    year={2023},\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports\n",
    "Import the required libraries and functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'numpy'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[4], line 5\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mconfigparser\u001b[39;00m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mos\u001b[39;00m\n\u001b[0;32m----> 5\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mevaluation\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mmetrics\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m createPR, recallAt100precision, recallAtK\n\u001b[1;32m      6\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mevaluation\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m show_correct_and_wrong_matches\n\u001b[1;32m      7\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mmatching\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m matching\n",
      "File \u001b[0;32m~/dengyong/VPR/VPR_Tutorial/evaluation/metrics.py:18\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m#   =====================================================================\u001b[39;00m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;66;03m#   Copyright (C) 2023  Stefan Schubert, stefan.schubert@etit.tu-chemnitz.de\u001b[39;00m\n\u001b[1;32m      3\u001b[0m \u001b[38;5;66;03m#\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[38;5;66;03m#   =====================================================================\u001b[39;00m\n\u001b[1;32m     17\u001b[0m \u001b[38;5;66;03m#\u001b[39;00m\n\u001b[0;32m---> 18\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mnumpy\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m \u001b[38;5;21;01mnp\u001b[39;00m\n\u001b[1;32m     21\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcreatePR\u001b[39m(S_in, GThard, GTsoft\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, matching\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mmulti\u001b[39m\u001b[38;5;124m'\u001b[39m, n_thresh\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m100\u001b[39m):\n\u001b[1;32m     22\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m     23\u001b[0m \u001b[38;5;124;03m    Calculates the precision and recall at n_thresh equally spaced threshold values\u001b[39;00m\n\u001b[1;32m     24\u001b[0m \u001b[38;5;124;03m    for a given similarity matrix S_in and ground truth matrices GThard and GTsoft for\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m     35\u001b[0m \u001b[38;5;124;03m    The integer n_tresh controls the number of threshold values and should be >1.\u001b[39;00m\n\u001b[1;32m     36\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'numpy'"
     ]
    }
   ],
   "source": [
    "import argparse\n",
    "import configparser\n",
    "import os\n",
    "\n",
    "from evaluation.metrics import createPR, recallAt100precision, recallAtK\n",
    "from evaluation import show_correct_and_wrong_matches\n",
    "from matching import matching\n",
    "from datasets.load_dataset import GardensPointDataset, StLuciaDataset, SFUDataset\n",
    "import numpy as np\n",
    "\n",
    "import ipywidgets as widgets\n",
    "from ipywidgets import interact, interact_manual, interactive\n",
    "from IPython.display import display\n",
    "\n",
    "from matplotlib import pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load dataset\n",
    "In this example, the input to a VPR algorithm are two image sets: the database DB and query Q (i.e., multi-set VPR). For later evaluation, we also load ground-truth information about correspondences. This ground truth only serves for evaluation and will neither be available nor required when deploying the algorithm.\n",
    "\n",
    "For demonstration, the relatively small *GardensPoint Walking* dataset with 200 images per image set, a subset of the *StLucia* dataset with 200 images can be loaded, or the 385-images dataset SFU Mountain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# Widget for selecting a dataset\n",
    "def select_dataset(Dataset=['GardensPoint', 'StLucia', 'SFU']):\n",
    "    return Dataset\n",
    "ds = interactive(select_dataset)\n",
    "display(ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# load dataset with ground truth\n",
    "dataset_name = ds.result\n",
    "if dataset_name == 'GardensPoint':\n",
    "    dataset = GardensPointDataset()\n",
    "elif dataset_name == 'StLucia':\n",
    "    dataset = StLuciaDataset()\n",
    "elif dataset_name == 'SFU':\n",
    "    dataset = SFUDataset()\n",
    "imgs_db, imgs_q, GThard, GTsoft = dataset.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Descriptor computation\n",
    "The main source of information about image correspondences are image descriptors. Local descriptors like DELF provide information for multiple regions of interest but are computationally expensive to compare. Holistic image descriptors reduce the computational complexity, but often have slightly lower performance. Feature aggregation methods such as HDC can be used to combine the local descriptors of an image in a single holistic descriptor vector.\n",
    "\n",
    "In the following, different holistic or local image descriptor can be selected. HDC-DELF, AlexNet-conv3 and NetVLAD are deep-learning based holistic descriptors. PatchNetVLAD is used as local descriptors in the following, so that it requires the highest runtime for creation and comparison."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# Widget for selecting a descriptor\n",
    "def select_descriptor(Descriptor=['HDC-DELF', 'AlexNet-conv3', 'NetVLAD', 'PatchNetVLAD']):\n",
    "    return Descriptor\n",
    "w = interactive(select_descriptor)\n",
    "display(w)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# select descriptor\n",
    "descriptor = w.result\n",
    "print(f'===== Compute {descriptor} descriptors')\n",
    "feature_extractor = None\n",
    "if descriptor == 'HDC-DELF':\n",
    "    from feature_extraction.feature_extractor_holistic import HDCDELF\n",
    "    feature_extractor = HDCDELF()\n",
    "elif descriptor == 'AlexNet-conv3':\n",
    "    from feature_extraction.feature_extractor_holistic import AlexNetConv3Extractor\n",
    "    feature_extractor = AlexNetConv3Extractor()\n",
    "elif descriptor == 'NetVLAD' or descriptor == 'PatchNetVLAD':\n",
    "    from feature_extraction.feature_extractor_patchnetvlad import PatchNetVLADFeatureExtractor\n",
    "    from patchnetvlad.tools import PATCHNETVLAD_ROOT_DIR\n",
    "    if w.result == 'NetVLAD':\n",
    "        configfile = os.path.join(PATCHNETVLAD_ROOT_DIR, 'configs/netvlad_extract.ini')\n",
    "    else:\n",
    "        configfile = os.path.join(PATCHNETVLAD_ROOT_DIR, 'configs/speed.ini')\n",
    "    assert os.path.isfile(configfile)\n",
    "    config = configparser.ConfigParser()\n",
    "    config.read(configfile)\n",
    "    feature_extractor = PatchNetVLADFeatureExtractor(config)\n",
    "\n",
    "# compute holistic descriptors\n",
    "if descriptor != 'PatchNetVLAD':\n",
    "    print('===== Compute reference set descriptors')\n",
    "    db_D_holistic = feature_extractor.compute_features(imgs_db)\n",
    "    print('===== Compute query set descriptors')\n",
    "    q_D_holistic = feature_extractor.compute_features(imgs_q)\n",
    "else:\n",
    "    print('=== WARNING: The PatchNetVLAD code in this repository is not optimised and will be slow and memory consuming.')\n",
    "    print('===== Compute reference set descriptors')\n",
    "    db_D_holistic, db_D_patches = feature_extractor.compute_features(imgs_db)\n",
    "    print('===== Compute query set descriptors')\n",
    "    q_D_holistic, q_D_patches = feature_extractor.compute_features(imgs_q)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Descriptor comparison and similarity matrix S\n",
    "To compare database and query descriptors, we either use the cosine similarity for holistic descriptors (e.g. computed by the inner product of the normalized descriptor vectors), or a more complex algorithmic approach for local descriptors. Although we might not want to compute the full similarity matrix S of all possible pairs in a large scale practical application, it can be useful for visual inspection purposes.\n",
    "\n",
    "In the following, the dense similarity matrix S between all descriptor pair is computed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# compare all descriptors and compute similarity matrix S\n",
    "print('===== Compute cosine similarities S')\n",
    "if descriptor != 'PatchNetVLAD':\n",
    "    # normalize descriptors and compute S-matrix\n",
    "    db_D_holistic = db_D_holistic / np.linalg.norm(db_D_holistic , axis=1, keepdims=True)\n",
    "    q_D_holistic = q_D_holistic / np.linalg.norm(q_D_holistic , axis=1, keepdims=True)\n",
    "    S = np.matmul(db_D_holistic , q_D_holistic.transpose())\n",
    "else:\n",
    "    S = feature_extractor.local_matcher_from_numpy_single_scale(q_D_patches, db_D_patches)\n",
    "\n",
    "# show similarity matrix S\n",
    "fig = plt.imshow(S)\n",
    "fig.axes.get_xaxis().set_visible(False)\n",
    "fig.axes.get_yaxis().set_visible(False)\n",
    "fig.axes.set_title('Similarity matrix S')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The image shows the similarity matrix $S{\\in}\\mathbb{R}^{|DB|\\times|Q|}$. In the GardensPoint Walking dataset, images with the same ID were recorded at same places, i.e., the i-th database image matches the i-th query image. Therefore, we can observe high similarities along the main diagonal of S."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Image matching\n",
    "The output of a VPR pipeline is typically a set of discrete matchings, i.e. pairs of query and database images. To obtain matchings for a query image from the similarity matrix, we can either find the single best matching database image (M1) or try to find all images in the database that show the same place as the query image (M2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# best matching per query in S for single-best-match VPR\n",
    "M1 = matching.best_match_per_query(S)\n",
    "\n",
    "# find matches with S>=thresh using an auto-tuned threshold for multi-match VPR\n",
    "M2 = matching.thresholding(S, 'auto')\n",
    "TP = np.argwhere(M2 & GThard)  # true positives\n",
    "FP = np.argwhere(M2 & ~GTsoft)  # false positives\n",
    "\n",
    "# show M's\n",
    "fig = plt.figure()\n",
    "ax1 = fig.add_subplot(121)\n",
    "ax1.imshow(M1)\n",
    "ax1.axis('off')\n",
    "ax1.set_title('Best match per query')\n",
    "ax2 = fig.add_subplot(122)\n",
    "ax2.imshow(M2)\n",
    "ax2.axis('off')\n",
    "ax2.set_title('Thresholding S>=thresh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both matrices $M{\\in}\\mathbb{R}^{|DB|\\times|Q|}$ show the matched images between the database and the query set. Left, only the best match per query was selected, leading to a thin line. Right, all image pairs with a similarity above a threshold were selected, s.t. we can also see multiple or no matches per query image."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# show correct and wrong image matches\n",
    "show_correct_and_wrong_matches.show(imgs_db, imgs_q, TP, FP)  # show random matches\n",
    "plt.title('Examples for correct and wrong matches from S>=thresh')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The green frame shows a correctly matched image pair, the red frame a wrongly matched image pair."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation\n",
    "To evaluate the quality of a similarity matrix S, we can apply a series of decreasing thresholds $\\theta$ to match more and more image pairs. Combined with ground-truth information, each threshold leads to a different set of true positives, false positives, true negatives and false negatives, which then provides one point on the precision-recall curve.\n",
    "\n",
    "In the following, the precision-recall curve and the area under the precision-recall curve is computed and visualized for *multi-match VPR*, i.e. all matches between each query image and the database have to be identified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# precision-recall curve\n",
    "P, R = createPR(S, GThard, GTsoft, matching='multi', n_thresh=100)\n",
    "plt.figure()\n",
    "plt.plot(R, P)\n",
    "plt.xlim(0, 1), plt.ylim(0, 1.01)\n",
    "plt.xlabel('Recall')\n",
    "plt.ylabel('Precision')\n",
    "plt.title('Result on GardensPoint day_right--night_right')\n",
    "plt.grid('on')\n",
    "plt.draw()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph shows the precision-recall curve. A curve closer to the upper right corner would represent better performance. Precision=1 means that no false positives (FP) were extracted. A recall=1 means that all same places were found."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# area under precision-recall curve (AUPRC)\n",
    "AUPRC = np.trapz(P, R)\n",
    "print(f'\\n===== AUPRC: {AUPRC:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The AUPRC performance ranges between 0 and 1 (higher is better)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# maximum recall at 100% precision\n",
    "maxR = recallAt100precision(S, GThard, GTsoft, matching='multi', n_thresh=100)\n",
    "print(f'\\n===== R@100P (maximum recall at 100% precision): {maxR:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The maximum recall at 100% precision ranges between 0 and 1 (higher is better)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe kernel failed to start as the Python Environment 'vpr (Python 3.10.4)' is no longer available. Consider selecting another kernel or refreshing the list of Python Environments."
     ]
    }
   ],
   "source": [
    "# recall@K\n",
    "Rat1 = recallAtK(S, GThard, GTsoft, K=1)\n",
    "Rat5 = recallAtK(S, GThard, GTsoft, K=5)\n",
    "Rat10 = recallAtK(S, GThard, GTsoft, K=10)\n",
    "print(f'\\n===== recall@K (R@K): R@1: {Rat1:.3f}, R@5: {Rat5:.3f}, R@10: {Rat10:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recall@K ranges between 0 and 1 (higher is better). The recall@K measures the rate of query images with at least one actually matching database image. Accordingly, the metric gets better with increasing K."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {
     "03b21a34bb224e91aad2ed63c126d731": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "1432910e9d08484facda671e58aff8ee": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_fd1418fc51974ad38e148bcd152481bc",
       "style": "IPY_MODEL_7d63107e73d149c0b8f9dd411e105764",
       "value": " 200/200 [00:27&lt;00:00,  7.35it/s]"
      }
     },
     "1be715b7b2c74aaf9d0eac51b0e9374f": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_a6a6f351c3914febae32932a2cec2ff8",
        "IPY_MODEL_d4531df87195466084f3aadf4909799b",
        "IPY_MODEL_1432910e9d08484facda671e58aff8ee"
       ],
       "layout": "IPY_MODEL_255dd791a0d1455998867754c56dbf04"
      }
     },
     "255dd791a0d1455998867754c56dbf04": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "3660e81e393d43aca19ec262c285ed25": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "DropdownModel",
      "state": {
       "_options_labels": [
        "HDC-DELF",
        "AlexNet-conv3"
       ],
       "description": "Descriptor",
       "index": 0,
       "layout": "IPY_MODEL_a40fb16a1e644c48b59bee11b877e5c6",
       "style": "IPY_MODEL_d3ec80f392844d79b5fce1f9113592a6"
      }
     },
     "38dd2c45d5bb4e6aa9fb0df1f50f9aeb": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "443ffa0932994fe589a283474705bbe0": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "4c8457eb7ab6474a8c878c916259e15e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_03b21a34bb224e91aad2ed63c126d731",
       "style": "IPY_MODEL_443ffa0932994fe589a283474705bbe0",
       "value": " 200/200 [00:58&lt;00:00,  3.32it/s]"
      }
     },
     "545ecdc7cdbd4c00a3dc37e9aaee2d17": {
      "model_module": "@jupyter-widgets/output",
      "model_module_version": "1.0.0",
      "model_name": "OutputModel",
      "state": {
       "layout": "IPY_MODEL_b464bc0eee0045e3b602efc8a0043c67"
      }
     },
     "6a4ee9c19e17419abd925a51faece4cf": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "7d63107e73d149c0b8f9dd411e105764": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "804cfa8b81404c3c933a31b97e682b20": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "layout": "IPY_MODEL_e05d8e5b5b8944959bc64277606d5822",
       "max": 200,
       "style": "IPY_MODEL_ff15e20f67b0499594bb4718a7b7ee24",
       "value": 200
      }
     },
     "8930b5cfed4e47ed903593184061e2ff": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "99865cd52e0d46538aa9de5c363f393e": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_c21ba3f4c72948a9ad3eea56e2d409e3",
       "style": "IPY_MODEL_38dd2c45d5bb4e6aa9fb0df1f50f9aeb",
       "value": "100%"
      }
     },
     "a3972486667445fa8eb0d17320f74025": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "a40fb16a1e644c48b59bee11b877e5c6": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "a6a6f351c3914febae32932a2cec2ff8": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLModel",
      "state": {
       "layout": "IPY_MODEL_6a4ee9c19e17419abd925a51faece4cf",
       "style": "IPY_MODEL_def03a8729274cbd8720f1412dcf04e4",
       "value": "100%"
      }
     },
     "aa15f3549c104d11ab42b6b24d4d1384": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "b464bc0eee0045e3b602efc8a0043c67": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "c21ba3f4c72948a9ad3eea56e2d409e3": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "d3ec80f392844d79b5fce1f9113592a6": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "DescriptionStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "d4531df87195466084f3aadf4909799b": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "FloatProgressModel",
      "state": {
       "bar_style": "success",
       "layout": "IPY_MODEL_a3972486667445fa8eb0d17320f74025",
       "max": 200,
       "style": "IPY_MODEL_d706e66f992a42018fa6961890b36e41",
       "value": 200
      }
     },
     "d706e66f992a42018fa6961890b36e41": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": ""
      }
     },
     "def03a8729274cbd8720f1412dcf04e4": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HTMLStyleModel",
      "state": {
       "description_width": "",
       "font_size": null,
       "text_color": null
      }
     },
     "e05d8e5b5b8944959bc64277606d5822": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "ebb736754c2f4a21960f0aa31d5b770c": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "VBoxModel",
      "state": {
       "_dom_classes": [
        "widget-interact"
       ],
       "children": [
        "IPY_MODEL_3660e81e393d43aca19ec262c285ed25",
        "IPY_MODEL_545ecdc7cdbd4c00a3dc37e9aaee2d17"
       ],
       "layout": "IPY_MODEL_8930b5cfed4e47ed903593184061e2ff"
      }
     },
     "fd1418fc51974ad38e148bcd152481bc": {
      "model_module": "@jupyter-widgets/base",
      "model_module_version": "2.0.0",
      "model_name": "LayoutModel",
      "state": {}
     },
     "fedf440ae87e4439a516ba44e4d1e6c7": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "HBoxModel",
      "state": {
       "children": [
        "IPY_MODEL_99865cd52e0d46538aa9de5c363f393e",
        "IPY_MODEL_804cfa8b81404c3c933a31b97e682b20",
        "IPY_MODEL_4c8457eb7ab6474a8c878c916259e15e"
       ],
       "layout": "IPY_MODEL_aa15f3549c104d11ab42b6b24d4d1384"
      }
     },
     "ff15e20f67b0499594bb4718a7b7ee24": {
      "model_module": "@jupyter-widgets/controls",
      "model_module_version": "2.0.0",
      "model_name": "ProgressStyleModel",
      "state": {
       "description_width": ""
      }
     }
    },
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
